Assignment 2

To be delivered until 2023/01/06 23:59:59.

1) Arduino

You will start by setting up a series of connections in order to extract some data with the Arduino. First make the connections as shown below. Mind the direction of the temperature sensor. If you have an incorrect position, you will be connection the power to the ground and vice-versa and you will damage the sensor. The photoresistor sensor on the other hand has no polarity.

On this problem, you will read temperature and luminance from the sensors and print them on the serial.

1) Code an Arduino sketch, where the value of temperature and luminance are printed to the serial. For each serial print that you make, print the value of temperature, then a semicolon, then the value of luminance with a new line (use no whitespaces). You can do this by using three separate Serial.print, with the last one being a Serial.println. Print values 5 times per second (use the delay function to control this). Manually influence the readings of the sensors, by covering the photoresistor or shining light on it, and by lightly and carefully touching the temperature sensor to increase its temperature readings.

Note that the temperature sensor appears not to be very reliable. Since the objective of this exercise is just to plot the results, this should not be an issue.

Copy and paste your arduino code below. You may use a python code cell, even though the code can not be run.

Hint: for the temperature value to be in celsius, divide the read value by 1024 and multiply it by 500. The luminance does not have to be converted

To import the data into Arduino, keep it running (the Serial Monitor must be closed in Arduino) and run the following code. Change the COM port to your own. This block of code will read 1000 values from the Serial. Given that each observation is taken every 0.2 seconds, it should take a minute and a half.

Convert the data into a pandas dataframe and save it in a csv file. Besides the value of temperature and luminance, also include the time, considering the first observation at $t=0$ and every observation 0.2 seconds after the previous one. The file must be submitted in Fenix and included in your Github repo.

Plot the Temperature against time, the luminance against time and the temperature against the luminance.

In the luminance plot, the value remain constant for the first twenty seconds, when the photoresistor sensor is exposed to the ambient light without any disturbances. Then, a flashlight is pointed to the sensor, thus the rise in value. Five seconds later, the sensor is covered and the luminance level drops. The flashlight is again pointed to the sensor and the value rises again. For a brief moment, the sensor is just exposed to the ambient light and finally is again covered.

In the temperature plot the values are constantly changing, since it is not very reliable. Also, it is exposed to ambient air so several disturbances may change the values. Only when it is covered (during a few seconds starting in the 25 second's mark and again in the 33 second's mark) it presents stable values.

In the third plot, a scatter plot, the different situations can be identified as is represented in the following image.

2) Databases

For the databases part of this assignment, you will use the mimic-iii database from the laboratory session. Start by adding a few new tables to the database, using the SQL files included in the assignment's files. Open PGAdmin and connect to your mimic-iii database. To properly load these tables, load the following files exactly and by the order presented.

1) Run demographic.sql

2) Run lab_firstday.sql

You will now have to answer a few SQL questions.

1. Open the connection to your mimic-iii database. If you want, you can delete your credentials before submitting the assignment, but if you do so, please run the notebook first, for the results to be displayed.

2. Create a function that receives an SQL query and automatically opens a cursor, queries the database, extracts the columns, creates a pandas database, and closes the connections.

3. Query the table admissions filtering for admission type as emergency and insurance as private.

4. Query the table admissions, filtering for the same conditions as the previous exercise (admission type as emergency and insurance as private). Join the "drgcodes" table on the admission ID. Display only the columns regarding the subject id, admission id, time of death, and description of the drug.

5.1. Obtain the dataset for this problem, by running the SQL query below.

5.2. Close the connection to your SQL server.

5.3. Prepare your dataset:

6. Fit the following tree-based classifiers to the dataset. For each method:

a. Decision tree.

b. Random forest

c. Gradient Boosting

7.1. Perform a grid search cross-validation on the Gradient boosting methods, changing the value of the learning rate (0.01 to 0.5) and the number of estimators (50-500). Consider the mean AUC of the folds as the performance measure.

7.2. Plot a scatterplot of the learning rate versus the number of estimators, with the mean AUC as the color gradient.

8.1. Perform forward stepwise selection on the dataset. Use the best parameters of the gradient boosting method obtained in 7.1..

8.2. Compare and comment the results from 8.1. with the features importance obtained through the grid search of queastion 7.1..

Conclusion

We can see that the two methods differ in the features selected. This may be the case since Forward stepwise selection is a greedy algorithm, meaning that it makes decisions based on the current state of the model without considering the long-term consequences. As a result, it may not always find the optimal combination of predictors.

3) Theoretical Questions

1. Consider a dataset where best subset, forward stepwise and backward stepwise selection will be performed. For each of the 3 approaches, we obtain $p+1$ models, $p$ being the total number of predictors. This means that each approach has a model with 0 predictors, one with 1 predictor, one with 2 predictor, up until one model with $p$ predictors. Answer and justify the following questions:

a) Which of the three models with $k, \, \forall_{k \in [0,p]}$ predictors has the smallest training RSS?

Best subset selection is a method for selecting a subset of predictors in a multiple regression model. It involves selecting the best model that includes all possible combinations of predictors. Forward stepwise selection is a method for selecting a subset of predictors in a multiple regression model. It involves starting with an empty model and iteratively adding one predictor at a time until all predictors are included in the model. Backward stepwise selection is a method for selecting a subset of predictors in a multiple regression model. It involves starting with a model that includes all predictors and iteratively removing one predictor at a time until only one predictor is left in the model.

The model that uses the best subset approach is the one that has the smallest training RSS, although forward stepwise and backward stepwise seletion may choose equally excellent models. The best subset selection method searches all possible models with k predictors, selecting the smallest training RSS, whereas the other two methods iteratively explore a subset starting with the best k-1 model and selecting the best k given a fixed k-1 or the reverse.

b) Which of the three models with $k, \, \forall_{k \in [0,p]}$ predictors has the smallest test RSS?

The model that uses the best subset approach is the one that has the smallest test RSS, given that it considers more models than the other approaches. However, it is probable to overfit and the others approaches can chose a model that fits the test data better.

c) Evaluate the following statements with true or false. Justify your answers.

i. The predictors in the k-variable model identified by forward stepwise selection are a subset of the predictors in the (k+1)-variable model identified by forward stepwise selection.

True. k+1 variable is the same as k variable with one predictor added.

ii. The predictors in the k-variable model identified by backward stepwise selection are a subset of the predictors in the (k + 1)-variable model identified by backward stepwise selection.

True. k variable is the same as k+1 variable with one predictor removed.

iii. The predictors in the k-variable model identified by backward stepwise selection are a subset of the predictors in the (k + 1)-variable model identified by forward stepwise selection.

False. Given that forward stepwise starts with an empty model and iteratively adds predictors and the backward stepwise starts with all the predictors and iteratively removes until the model is empty, the path isn't the same. Thus, usually the predictors in the k variable model identified by backward stepwise aren't a subset of the predictors in the k+1 variable model identified by forward stepwise. However, this could happen for a particular example.

iv. The predictors in the k-variable model identified by forward stepwise selection are a subset of the predictors in the (k+1)-variable model identified by backward stepwise selection.

False. Given that forward stepwise starts with an empty model and iteratively adds predictors and the backward stepwise starts with all the predictors and iteratively removes until the model is empty, the path isn't the same. Thus, usually the predictors in the k variable model identified by forward stepwise aren't a subset of the predictors in the k+1 variable model identified by backward stepwise. However, this could happen for a particular example.

v. The predictors in the k-variable model identified by best subset selection are a subset of the predictors in the (k + 1)-variable model identified by best subset selection.

False. It is not guaranteed that it will choose the same predictors.

2. Ridge regression tends to give similar coefficient values to correlated variables, whereas lasso regression may give substantially different coefficients to correlated variables. This questions explores this property in a simplified setting.

Suppose that $n=2$, $p=2$, $x_{11} = x_{12}$, $x_{21} = x_{22}$. Moreover, suppose that $y_1 + y_2 = 0$ and $x_{11} + x_{21} = 0$ and $x_{12} + x_{22} = 0$, meaning that the estimate for the intercept in a least squares, ridge regression, or lasso regression is zero: $\hat{\beta} = 0$.

a) Write the ridge regression optimization problem in this setting.

3.2a.png

b) Prove that in this setting, the ridge regression coefficient estimates satisfy $\hat{\beta}_1 = \hat{\beta}_2$.

3.2b.png

c) Write the lasso regression optimization problem in this setting.

3.2c.png

d) Prove that in this setting, the lasso regression coefficients $\hat{\beta}_1$ and $\hat{\beta}_2$ are not unique, meaning that there are many possible solutions to the optimization problem in (c). Describe these solutions.

3.2d.png

3. Draw an example of a partition of two-dimensional feature space that could result from recursive binary splitting. Your example should contain at least six regions. Draw a decision tree corresponding to this partition. Be sure to label all aspects of your figures, including the regions R1, R2,..., the cutpoints t1, t2,..., and so forth.

If you prefer you can draw it by hand or in any software and use a scan of it.

3.3.png

4. In 2 dimensions, a linear decision boundary takes the form $\beta_0 + \beta_1 X_1 + \beta_2 X_2 = 0$. Consider a nn-linear decision boundary:

a) Sketch the curve

$$(1 + X_1)^2 + (2 - X_2)^2 = 4$$

Additionally, indicate on your sketch the set of points that verify the condition

$$(1 + X_1)^2 + (2 - X_2)^2 > 4$$

and the condition

$$(1 + X_1)^2 + (2 - X_2)^2 \leq 4$$

b) Suppose that a classifier assigns an observation to the blue class if $(1 + X_1)^2 + (2 - X_2)^2 > 4$ and to the red class otherwise. To what class are the following observations classified? (0,0), (-1,1), (2,2), (3,8)

c) Prove that while the decision boundary in (b) is not linear in terms of $X_1$ and $X_2$, it is linear in terms of $X_1$, $X_1^2$, $X_2$, and $X_2^2$.

To prove that, we want to find a,b,c,d,e such that: $a*X1$ + $b*X1^2$ + $c*X2$ + $d*X2^2$ + $e$ $\leq 0$

By expanding the provided equation we get: $X1^2$ + $2*X1$ + $X2^2$ - $4*X2$ - $6$ $\leq 0$

So we get: $a = 1$, $b = 2$, $c = 1$, $d = -4$, $e = -6$

4) Laboratory Questions

What are the advantages and disadvantages of relational dabases versus graph databases, and when should one type be preferred over the other?

The main difference between these two types of databases is how they store relationships between entities.

In a relational database, data is organized into tables made up of rows and columns. Each row, or record, contains all the information about a specific item, and each column represents a specific piece of information about that item.

Graph databases are designed for storing and processing data with intricate relationships between entities. Rather than using tables to organize data, graph databases represent data using nodes and edges, which depict the connections between data points.

Relational databases are faster, if a significant number of records is stored. Therefore, they tend to have a smaller memory footprint

When dealing with heavy relationship use cases, graph databases are better suited. They are particularly useful for analyzing intricate data sets, thus being a better choice when intricate relationships have to be found, such as recomendation engines, fraud detection or social media.

Graph databases should be used if:

Relational databases should be used if:

The main advantages and disadvantages are:

Thus, relational databases are preferred when the data is structured and the queries are complex, while graph databases are preferred when the data is highly connected and the queries involve traversing relationships.

For example, when building a social network, the profile information of user, such as age or sex could be saved using a relatioanal database. When saving the information about wihch friends are friends with who, a graph databse could be more suitable since you might create an edge between two user nodes to represent that they are friends instead of storing all the connections for all users using a table structure like in a relational database.